1. Field of the Invention
The present invention relates to the protection of computer systems. More particularly, the present invention relates to a method and apparatus for detecting attempted exploitation of a computer system by malicious code, such as by a buffer overflow.
2. Description of Related Art
Often code written by programmers does not perform proper bounds checking when reading data from a file or an external source leaving it vulnerable to buffer overflow. Due in part at least to the large number of potentially vulnerable computer systems, buffer overflow is one of the most common forms of security exploits.
Buffers are memory areas, which generally hold a predefined amount of finite data. A buffer overflow occurs when a program attempts to store data into a buffer, where the data is larger than the size of the buffer. When the data exceeds the size of the buffer, the extra data can overflow into the adjacent memory locations. In this manner, it is possible to corrupt valid data and possibly to change the execution flow and instructions in the stack.
Two buffers commonly exploited by buffer overflows are the stack and heap. A stack is a static region of memory utilized by a program to hold local variables, return addresses, and data used by the program's subroutines. A heap is a dynamically allocated region of memory for use by a program.
By exploiting a buffer overflow in these memory areas, it is possible to inject malicious code, sometimes called malicious shellcode, into the execution flow. This shellcode allows remote system level access, giving unauthorized access to not only malicious hackers, but also to replicating malicious code, e.g., computer worms.